ENH: select_dypes impl #7434

cpcloud · 2014-06-11T18:22:54Z

examples:

In [8]: paste
   df = DataFrame({'a': list('abc'),
                   'b': list(range(1, 4)),
                   'c': np.arange(3, 6).astype('u1'),
                   'd': np.arange(4.0, 7.0),
                   'e': [True, False, True],
                   'f': [False, True, False],
                   'g': pd.date_range('now', periods=3).values})
   df['h'] = df.g.diff()
   df['i'] = np.arange(3, 6).astype('u8')
   df['j'] = pd.date_range('20130101', periods=3).values
   df
## -- End pasted text --
Out[8]:
   a  b  c  d      e      f                   g      h  i          j
0  a  1  3  4   True  False 2014-06-22 23:40:53    NaT  3 2013-01-01
1  b  2  4  5  False   True 2014-06-23 23:40:53 1 days  4 2013-01-02
2  c  3  5  6   True  False 2014-06-24 23:40:53 1 days  5 2013-01-03

In [9]: paste
   df.select_type(include=[bool])
## -- End pasted text --
Out[9]:
       e      f
0   True  False
1  False   True
2   True  False

In [10]: paste
   df.select_type(include=['number', 'bool'], exclude=['unsignedinteger'])
## -- End pasted text --
Out[10]:
   b  d      e      f      h
0  1  4   True  False    NaT
1  2  5  False   True 1 days
2  3  6   True  False 1 days

In [11]: np.timedelta64.mro()  # this is an integer type
Out[11]:
[numpy.timedelta64,
 numpy.signedinteger,
 numpy.integer,
 numpy.number,
 numpy.generic,
 object]

In [13]: paste
   df.select_type(include=['object'])
## -- End pasted text --
Out[13]:
   a
0  a
1  b
2  c

jreback · 2014-06-11T18:34:07Z

should not allow ndim=1 and your indexer for ndim>2 is not right, (its a bit tricky with Panels), as it depends on the orientation to figure out which dtype you'd be getting, so would just disallow anything but ndim==2 for now

cpcloud · 2014-06-11T18:45:51Z

Yep that's the wip part :)

cpcloud · 2014-06-11T18:46:12Z

I'll just implement on frame directly

jreback · 2014-06-11T19:03:03Z

This is a subtle issue. specifying float should match on the kind of the type rather than `==``

In [1]: np.dtype('float') == 'float32'
Out[1]: False

In [2]: np.dtype('float') == 'float64'
Out[2]: True

In [3]: np.dtype('float') == 'float'
Out[3]: True

need some tests with string dtypes too (and invalid ones I think you should raise), e.g. datetime64[D]. should raise, but datetime and datetime64 I think ok (just because the first is TOO specific)

cpcloud · 2014-06-11T19:09:09Z

I check the dtype.type attribute if it's a subclass of the passed in dtype. All dtypes are converted using np.dtype before any checks occur

cpcloud · 2014-06-11T19:56:53Z

whew jeez there are all sorts of dtype special cases

cpcloud · 2014-06-11T20:07:57Z

@jreback i think 'integer' and 'int' should mean different things to be consistent with numpy; i'm also not really too keen on 'numeric' since that's not a dtype, but i'm allowing 'number'

cpcloud · 2014-06-11T20:08:35Z

not sure i see the problem with datetime64[D] even tho we don't really allow it

hayd · 2014-06-11T20:19:11Z

pandas/core/generic.py

+
+        # empty include/exclude -> defaults to True
+        wanted = pd.Series(dict.fromkeys(index, not bool(include)))
+        not_wanted = pd.Series(dict.fromkeys(index, not bool(exclude)))


FYI you can just do pd.Series(not bool(exclude), index). (Semantically this one seems strange, I think this is actually not_not_wanted?? :s )

yeah the implementation is a bit unreadable right now ... i'm working on it

No worries. This'll be neat!

jreback · 2014-06-22T12:38:12Z

hows this coming?

cpcloud · 2014-06-22T14:35:54Z

Will get to it today. Work has picked up in the last week or so, haven't had as much time as I would like for pandas. :(

cpcloud · 2014-06-22T18:25:08Z

@jreback should empty include and empty exclude return an empty frame?

cpcloud · 2014-06-22T18:26:02Z

or how about a raise since that doesn't really make any sense

cpcloud · 2014-06-22T18:26:20Z

oh duh i'm already doing that nevermind

jreback · 2014-06-22T18:27:23Z

yeh I think you have to have either include or exclude non empty / not none

still could return empty frame of course

cpcloud · 2014-06-22T19:08:23Z

now disallowing datetime64[D] and others like it, i agree with you @jreback that it doesn't make sense ... allowing ns tho, since that's our main representation

jreback · 2014-06-22T19:31:14Z

you could allow thinks like

'datetime', 'date time', datetime.datetime, np.datetime

eg generic datetime dtypes (you may need to have a dict/regex matcher for things like this)

cpcloud · 2014-06-22T20:30:41Z

i'm already allowing 'datetime', 'datetime64' and 'datetime64[ns]', in addition to the dtype form of the last two... i think that's enough.

jreback · 2014-06-22T20:48:55Z

ok cool

cpcloud · 2014-06-22T21:33:36Z

the one thing that's currently not possible is getting columns that are non string object dtypes

cpcloud · 2014-06-22T21:33:50Z

ie if you ask for object you'll always get strings as well as any other objects

jreback · 2014-06-22T21:34:46Z

no distinction between string and those anyhow (their could be but not now)

jreback · 2014-06-26T14:11:32Z

doc/source/indexing.rst

@@ -1149,11 +1143,97 @@ default value.
   s.get('a')               # equivalent to s['a']
   s.get('x', default=-1)

+Selecting columns based on ``dtype``


I think this should be here: (the whole section): http://pandas-docs.github.io/pandas-docs-travis/basics.html#dtypes

ok will move

cpcloud · 2014-06-27T19:39:33Z

@jreback @jorisvandenbossche any more comments?

jreback · 2014-06-27T19:54:24Z

pandas/core/common.py

@@ -1603,6 +1603,66 @@ def _get_fill_func(method):
 #----------------------------------------------------------------------
 # Lots of little utilities

+def _validate_date_like_dtype(dtype):
+    try:
+        typ = np.datetime_data(dtype)[0]


never knew that was a method like this!

yeah only thing is that the second tuple element is num which is used only internall in numpy AFAICT, it seems to always be 1 when used in python but i think it's used to do conversions between different units.

you can also just test the kind as M right? (

no i'm checking that the unit is either generic (this occurs when a user passes in np.datetime64; 'datetime64[generic]' shouldn't really be passed in by a user, tho it will work) or ns, i could write something simple that parses out the unit for us, but not sure that's necessary

jreback · 2014-07-01T15:53:50Z

I think you just need a couple of doc edits (from above).....

cpcloud · 2014-07-02T14:44:45Z

Yeah sorry things just picking up a lot at work. Will do this today, so we can get the ball rolling on the release.

cpcloud · 2014-07-06T13:00:55Z

@jreback good 2 go?

jreback · 2014-07-06T13:25:25Z

do you test if df.select_dtypes(['foo']) raises? e.g. a non-dtype (or do you just ignore this?) I think should raise

cpcloud · 2014-07-06T13:27:09Z

good call, don't think i test thits

jreback · 2014-07-07T13:13:49Z

@cpcloud when you have a chance

cpcloud · 2014-07-07T16:04:04Z

@jreback this is good to go i think

jreback · 2014-07-07T16:06:01Z

doc/source/basics.rst

+
+
+
+Working with package options


is this supposed to be added?

crap totally missed that htanks

maybe you are not rebased to master?

as soon as you are ready then go ahead and merge (don't worry about travis). it already passed :)

jreback · 2014-07-07T16:18:22Z

I think this section slipped back in somehow: http://pandas-docs.github.io/pandas-docs-travis/options.html (as the new options section is there)

jreback · 2014-07-07T16:20:05Z

hmm. go ahead and merge this. let's fix in another commit (removing the options stuff from basics) https://github.com/pydata/pandas/pull/7578/files

jreback · 2014-07-07T16:21:58Z

never mind, master looks fine

cpcloud · 2014-07-07T16:22:15Z

cool merging

ENH: select_dypes impl

jreback · 2014-07-07T16:28:07Z

nice !

jreback · 2014-07-07T16:28:52Z

I think you can add a mini example in 0.14.1 for select_dtypes (can be same as from docs).....

jreback · 2014-07-07T18:38:04Z

you can now update SO with your docs link: http://pandas-docs.github.io/pandas-docs-travis/basics.html#basics-selectdtypes for the gazillion questions that always happen about this :)

jankatins · 2014-07-09T20:18:34Z

This needs to be tested with dtype "category" #7217

jreback · 2014-07-09T20:53:14Z

jreback@a43c6c0

makes this work

cpcloud added API Design labels Jun 11, 2014

cpcloud added this to the 0.14.1 milestone Jun 11, 2014

cpcloud self-assigned this Jun 11, 2014

hayd reviewed Jun 11, 2014
View reviewed changes

jreback reviewed Jun 26, 2014
View reviewed changes

jreback changed the title ~~WIP: select_type impl~~ WIP: select_dypes impl Jun 26, 2014

cpcloud changed the title ~~WIP: select_dypes impl~~ ENH: select_dypes impl Jun 27, 2014

jreback reviewed Jun 27, 2014
View reviewed changes

jreback reviewed Jul 7, 2014
View reviewed changes

WIP: select_dtypes impl

4fc5ae7

cpcloud added a commit that referenced this pull request Jul 7, 2014

Merge pull request #7434 from cpcloud/with-types

bcbc7af

ENH: select_dypes impl

cpcloud merged commit bcbc7af into pandas-dev:master Jul 7, 2014

cpcloud deleted the with-types branch July 7, 2014 16:22

jreback mentioned this pull request Jul 9, 2014

WIP: categoricals as an internal CategoricalBlock GH5313 #7217

Merged

31 tasks

jseabold mentioned this pull request Jul 21, 2015

Add _get_numeric_data() to the public API? #5775

Closed

ENH: select_dypes impl #7434

ENH: select_dypes impl #7434

Conversation

cpcloud commented Jun 11, 2014

jreback commented Jun 11, 2014

cpcloud commented Jun 11, 2014

cpcloud commented Jun 11, 2014

jreback commented Jun 11, 2014

cpcloud commented Jun 11, 2014

cpcloud commented Jun 11, 2014

cpcloud commented Jun 11, 2014

cpcloud commented Jun 11, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 22, 2014

cpcloud commented Jun 22, 2014

cpcloud commented Jun 22, 2014

cpcloud commented Jun 22, 2014

cpcloud commented Jun 22, 2014

jreback commented Jun 22, 2014

cpcloud commented Jun 22, 2014

jreback commented Jun 22, 2014

cpcloud commented Jun 22, 2014

jreback commented Jun 22, 2014

cpcloud commented Jun 22, 2014

cpcloud commented Jun 22, 2014

jreback commented Jun 22, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Jun 27, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jul 1, 2014

cpcloud commented Jul 2, 2014

cpcloud commented Jul 6, 2014

jreback commented Jul 6, 2014

cpcloud commented Jul 6, 2014

jreback commented Jul 7, 2014

cpcloud commented Jul 7, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jul 7, 2014

jreback commented Jul 7, 2014

jreback commented Jul 7, 2014

cpcloud commented Jul 7, 2014

jreback commented Jul 7, 2014

jreback commented Jul 7, 2014

jreback commented Jul 7, 2014

jankatins commented Jul 9, 2014

jreback commented Jul 9, 2014